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PORT Adapter for High-Bandwidth Bus 

CROSS-REFERENCE TO RELATED APPLICATIONS 
[0001] This application is related to prior co-pending commonly assigned application 
Ser. No. 09/790,970, filed February 22, 2001, entitled "Apparatus and technique for 
conveying per-channel flow control information to a forwarding engine of an intermediate 
network node," of Guy Fedorkow et al, the entire contents of which are hereby incorporated 
by reference as if fully set forth herein. 

COPYRIGHT NOTICE 
[0002] A portion of the disclosure of this patent document contains material which is 
subject to copyright protection. The copyright owner has no objection to the facsimile 
reproduction of the patent disclosure, as it appears in the Patent & Trademark Office patent 
file or records, but otherwise reserves all copyright rights whatsoever. Copyright © 2003 
Cisco Systems, Inc. 

CD-ROM APPENDIX 

[0003] This disclosure includes a CD-ROM Appendix consisting of 134 pages in one file 
on one CD-ROM, the entire contents of which are hereby incorporated by reference as if 
fully set forth herein. 

FIELD OF THE INVENTION 
[0004] This invention generally relates to digital computer systems, and relates more 
particularly to digital computers that include a SPI-4 bus. 
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BACKGROUND OF THE INVENTION 
[0005] The approaches described in this section could be pursued, but are not necessarily 
approaches that have been previously conceived or pursued. Therefore, unless otherwise 
indicated herein, the approaches described in this section are not prior art to the claims in this 
application and are not admitted to be prior art by inclusion in this section. 
[0006] Digital computers use input/output (I/O) buses for transferring information 
between peripheral devices and a computer central processing unit and computer memory. 
I/O functions are also required in systems with multiple distributed processors and multiple 
distributed memories. 

[0007] A variety of I/O bus architectures are used in such computer systems, including 
Personal Computer Interface (PCI). The System Packet Interface-4 bus is a relatively new 
high-bandwidth bus that is generally used in data packet processing systems for computer 
networks, such as high-speed routers and switches. Characteristics of the SPI4.2 bus 
architecture are described in an Interface Specification that is available in the document 
www.oiforum.com/public/impagreements.html. In this document, the term "SPI-4" is 
equivalent to "SPI4.2," and includes variants and equivalents of the SPI4.2 bus architecture. 
[0008] Although the SPI-4 bus provides a high-speed communication path for packet 
data within a computer system, the SPI-4 bus is not suitable for direct conmiunication to 
external networks or devices. Interfacing a host with a SPI-4 bus to a network normally 
requires providing logical or physical ports or interfaces that are coupled to other devices or 
networks. Some port adapters are architected as service adapters that have no ports or 
interfaces, but provide a particular kind of packet processing service for a host, such as 
compression or decompression, encryption or decryption, etc. 



50325-0790 (Seq. No. 7683) 



-2- 



[0009] Users and manufacturers particularly desire to have host systems that can 
accommodate ports and interfaces that use different technologies, such as Ethernet, Fast 
Ethemet, Gigabit Ethemet, optical, serial or other interfaces. In one approach, a host router or 
switch is hard-wired with a variety of different ports. However, a user cannot re-configure 
such a host if the user's port requirements change. Such users and manufacturers want to 
have a host system that is adaptable to changing port and interface requirements. 
[0010] Hot swapping may also damage some devices connected to the SPI-4 bus such as 
devices using Complimentary Metal Oxide Semiconductor (CMOS) technology. CMOS 
devices are exposed to large currents when inputs to CMOS receivers are within the CMOS 
switching region. Some CMOS receivers have two field effect transistors (FETs) connected 
in series with a first FET connected to a positive power supply rail and a second FET 
connected to a negative power supply rail. When the input to the two FETS is in the 
switching region, both FETs can be continuously tumed on at the same time creating a DC 
current path directly through the CMOS device. The continuous on state of the two FETs can 
dissipate enough power to damage the CMOS device. 

[0011] CMOS devices also experience latch-up conditions when an input is driven 
beyond one of the CMOS power supply rails. In the latch-up condition, parasitic transistors 
in the CMOS structure dissipate large amounts of power that can destroy the CMOS device. 
Both power dissipation conditions described above can result from hot swapping on the host 
interface bus. 

[0012] U.S. Pat. No. 5,793,987 and U.S. Pat. No. 6,163,824 of Quackenbush et al. 
disclose a port adapter with separate PCI local bus and local bus, and associated processing 
methods. A port adapter is an electronic device that provides one or more ports and that plugs 
into a host system to provide additional features or functions for the host. The technology of 
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Quackenbush et al. has been used in PCI bus-based port adapters in the Cisco 7200 Series 
Routers and Cisco 7500 Series Routers, from Cisco Systems, Inc., San Jose, California. 
However, the technology of Quackenbush et al. is not suitable for hosts having SPI-4 bus 
architectures because of vast technical differences between the PCI bus and the SPI-4 bus. 
For example, the PCI bus cannot process data that is arriving from interfaces at high rates 
such as 10 gigabits per second (Gbps). 

[0013] Still another drawback of existing port adapters is that they do not interoperate 
seamlessly with heterogeneous network environments. For example, a host with a plurality of 
port adapters may communicate with external networks or devices using any of a large 
number of network technologies. As a result, data packets that are received at the port 
adapters may have any of a large nimiber of different formats. Requiring the host system to 
understand and process a large number of different packet formats would be complicated and 
lack scalability to new technologies. Further, it would be impractical to have one generic 
packet format used between each type of port adapter and the host system, because of 
differences in the type and quantity of data carried in packets of different technologies. 
[0014] Thus, there is a need for a port adapter that can process a particular packet format 
for a particular technology, and provides data to the host in a single consistent packet format 
for internal processing. 

[00151 Based on the foregoing, there is a clear need in the relevant technical field for a 
port adapter that can interface a host system having a SPI-4 bus architecture to different 
network technologies. More broadly, there is a need for an apparatus that can provide a hot- 
pluggable adaptive interface from the SPI-4 bus of a host to external peripheral equipment. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

[0016] The present invention is illustrated by way of example, and not by way of 
limitation, in the figures of the accompanying drawings and in which like reference numerals 
refer to similar elements and in which: 

[0017] . FIG. 1 is a block diagram that illustrates an overview of a host system with one or 
more port adapters; 

[0018] FIG. 2 is a block diagram that illustrates an overview of a port adapter for the 
SPMbus; 

[0019] FIG. 3 is a more detailed block diagram of the port adapter of FIG. 2, according to 
one embodiment; 

[0020] FIG. 4A is a flow diagram of a process of adapting the operational behavior of 
SPI-4 bus of a host system based on a capability of a port adapter; 
[0021] FIG. 4B is a flow diagram that illustrates an overview of a process of 
transforming received data packets; 

[0022] FIG. 5 is a block diagram of a transformed packet format. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

[0023] A hot-pluggable port adapter for a high-speed bus is described. In the following 

description, for the purposes of explanation, numerous specific details are set forth in order to 

provide a thorough understanding of the present invention. It will be apparent, however, to 

one skilled in the art that the present invention may be practiced without these specific 

details. In other instances, well-known structures and devices are shown in block diagram 

form in order to avoid'unnecessarily obscuring the present invention. 

[0024] Embodiments are described herein according to the following outline: 

1 .0 General Overview 

2.0 Structural and Functional Overview 

2. 1 Port Adapter Architecture 

2.2 Adaptation to Different SPI-4 Operating Speeds 

2.3 Extended Flow Control Bus 

2.4 Pre-Processing Packets with Port Adapter 
3.0 "Implementation Mechanisms — ^Hardware Overview 
4.0 Extensions and Altematives 

1.0 GENERAL OVERVIEW 

[0025] The needs identified in the foregoing Background, and other needs and objects 
that will become apparent firom the following description, are achieved in the present 
invention, which comprises, in one aspect, a hot-pluggable port adapter for connecting 
network interfaces to a host system through a SPI-4 bus. The port adapter conmiunicates with 
the host system through a port adapter/host interface that includes the SPI-4 bus and a control 
bus; an extended flow control bus may be provided. Methods are provided for selecting and 
using one of a small plurality of different packet formats for various networking 
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technologies, so that the port adapter can hide details of the technology that it handles from 
the host system, and for operating the host system's SPI-4 bus at one of several speeds based 
on bandwidth requirements of the port adapter. 

[0026] According to one aspect, the invention provides a port adapter for coupling zero 
or more network interfaces to a host system having a SPI-4 bus, the port adapter comprising 
zero or more network interfaces; a SPI-4 bus coupled to a host system to provide a 
communication channel between the host and the network interfaces; a control bus coupled 
to the host system for controlling and monitoring the port adapter; and interface logic that 
interfaces the SPI-4 bus and the control bus to the network interfaces. 
[0027] According to one feature, the interface logic comprises a field programmable gate 
array (FPGA), application-specific integrated circuit (ASIC), a combination of these and one 
or more other hardware elements, or a combination of one or more other hardware elements. 
According to another feature, an identity bus is coupled to the host system to allow 
identification of the port adapter by the host system. In yet another feature, an extended flow 
control bus is provided on which the port adapter may convey FIFO status to the host system 
on a plurality of separate logical channels. In still another feature, a clock bus communicates 
network timing information between a port and the host system, for providing 
synchronization of a port to another port, synchronization of a host systeiri reference 
oscillator to a port, or synchronization of a port to a reference clock that is extemal to the 
host system. 

[0028] The port adapter may further comprise a power control circuit that selectively 
generates power for the adapter during on Une insertion and removal of the port adapter from 
the host system while the host system remains powered on. According to one feature, an 
identification repository stores a unique identifier of a type of the port adapter. In a related 
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feature, the identification repository further stores one or more configuration parameter 
values associated with the port adapter. In yet another related feature, the identification 
repository comprises an electrically erasable progranmiable read only memory. The 
identification repository may store values that allow the host to determine whether the port 
adapter can be supported by the host system. The identification repository may store values 
that allow the host to determine one or more operating frequencies of the SPI-4 bus. 
[0029] hi another feature, an extended flow control bus is coupled to the host system that 
enables the adapter to communicate information on the SPI-4 bus using more than the 256 
logical chaimels that are conventionally available using the SPI-4 bus. In a related feature, 
flow control interface logic for the flow control bus comprises a calendar-based mechanism 
that allows the port adapter to convey buffer fill status of thousands of logical channels to the 
host system. The extended flow control bus may comprise a TDM calendar firame sync 
signal, a flow control clock signal, a status signal, and a parity signal. 
[0030] According to one feature, the adapter comprises zero interfaces, and wherein the 
interface logic is configured to receive one or more packets fi'om the host system, transform 
the packets according to a specified function, and send the transformed packets to the host 
system. In a related feature, the specified fimction comprises encryption or decryption. 
[0031] In another aspect, the invention provides a method of selectively determining an 
operating firequency for a SPI-4 bus of a host computer system that uses a port adapter, 
wherein the operating fi*equency may be different than a conventional SPI-4 bus operating 
fi-equency, comprising the steps of issuing a query from a host computer system having a 
SPI-4 bus to a port adapter, the port adapter comprising a SPI-4 bus that can be coupled to a 
host system for control and data between the host and the SPI-4 device, a control bus coupled 
in parallel with the SPI-4 bus between the host system and the port adapter for the port 
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adapter independently from the SPI-4 bus, and interface logic that interfaces the SPI-4 bus 
and the control bus to one of a plurality of line interfaces, and an identification repository; 
receiving, from the identification repository, an identification of the port adapter; 
determining, based on the information received from the identification repository, whether 
the host system SPI-4 bus can operate at a frequency that is compatible with at least one of 
the SPI-4 bus operating frequencies that are supported by the port adapter; and setting an 
operating frequency of the host system SPI-4 bus equal to a selected one of the SPI-4 bus 
operating frequencies that are supported by the port adapter. 

[0032] In one feature of this aspect, the setting step comprises setting the operating 
frequency of the host system SPI-4 bus equal to a fastest one of the SPM bus operating 
frequencies that are supported by the port adapter. In another feature, the method includes 
powering-on the port adapter only when the host system SPI-4 bus can operate at a frequency 
that is compatible with at least one of the SPI-4 bus operating frequencies that are supported 
by the port adapter. In a related feature, the method comprises powering-on the port adapter 
only when one or more factors are satisfied, wherein the factors are selected from the set 
consisting of: the host system has software support for a packet format required by the port 
adapter; the port adapter dissipates less than a maximum amount of power dissipation 
allowed by the host system; the host system can match a bandwidth required by the port 
adapter; or a license authorization requirement associated with the port adapter allows the 
port adapter to run on the host system. 

[0033] In another feature, the method fiirther comprises receiving, from the identification 
repository, values that allow the host to determine whether the port adapter can be supported 
by the host system, and one or more operating frequencies of the SPI-4 bus. The method may 
fiirther comprise receiving, from the identification repository, values that allow the host to 
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determine a packet format of data that is sent across the SPI-4 bus by the port adapter. In a 
related feature, the method may further comprise receiving, from the identification 
repository, one or more values specifying a packet format of data that is sent across the SPI-4 
bus by the port adapter. 

[0034] In yet another aspect, the invention provides a port adapter for coupling zero or 
more network interfaces to a host system having a SPI-4 bus, the port adapter comprising: 
zero or more network interfaces; a SPI-4 bus coupled to a host system to provide a 
communication channel between the host and the network interfaces; a control bus coupled 
to the host system for controUing and monitoring the port adapter; interface logic that 
interfaces the SPI-4 bus and the control bus to the network interfaces; and packet processing 
logic for pre-processing packets received on the interfaces by performing the steps of: 
receiving a first packet on an ingress interface of the port adapter; creating a second packet 
that conforms to a selected one of the intemal packet formats; transforming data from one or 
more fields of the first packet to one or more corresponding fields of the second packet; 
providing the second packet to a host system. 

[0035] In one feature of this aspect, the packet processing logic further comprises the 
steps of moving a remainder of a packet header and packet body from the first packet into the 
second packet. The packet processing logic may be configured to perform the step of 
selecting one of a plurality of intemal packet formats. The ingress interface may be, for 
example, an Ethernet interface, ATM interface, frame relay, serial interface, highly 
channelized interface, RPR interface, or POS interface, or any other interface now known or 
invented hereafter. 
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[0036] In other aspects, the invention encompasses a computer apparatus and a 
computer-readable medium configured to carry out the foregoing steps. 
2.0 STRUCTURAL AND FUNCTIONAL OVERVIEW 

2.1 PORT ADAPTER ARCHITECTURE 
[0037] A pluggable port adapter is used to connect zero or more ports or interfaces to a 
host system through a SPI-4 bus to add functionality to the host system. Typically the ports 
or interfaces are high-bandwidth optical ports or interfaces. The port adapter conraiunicates 
with the host system through a port adapter/host interface that includes the SPI-4 bus, a 
control bus, an extended flow control bus, and other signals and power lines. The ports or 
interfaces are configured and communicate in a normal manner over the SPI-4 bus while 
other functionality on the port adapter is controlled independently through the control bus. 
[0038] In this document, the term "SPI-4" is equivalent to "SPI4.2" and includes variants 
of the SPI4.2 bus architecture. Thus, an embodiment may use a bus that strictly adheres to 
the SPI-4 specification, or may use variants, enhancements, modifications or improvements 
to the SPI-4 specification. 

[0039] The host system uses a specialized identity bus to determine the identity of a port 
adapter, which the host system then uses to determine what programming and configuration 
is required. The control bus is used by the host system for configuring and controlling 
devices on the port adapter, and for updating programmable circuitry on the port adapter such 
as field programmable gate arrays (FPGAs). Alternatively, a JTAG bus can be used to update 
such programmable devices. FPGAs with volatile program memory must be programmed 
each time they are powerejd up and can be reprogrammed in the field by the host system to 
repair bugs and to enhance perforaiance and/or functionality. 
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[0040] In one embodiment, the SPI-4 bus in the port adapter is coupled to SPI-4 
temiination logic, which is coupled to one or more network interfaces, such as a framer, 
ATM SAR, etc. In cooperation, the SPI-4 termination logic and network interfaces control 
zero or more ports, which are coupled to zero or more communication lines, devices, or 
networks such as local area networks (LAN) and wide area networks (WAN). The SPM 
termination logic and network interfaces cooperate to receive data from the ports or 
interfaces and then retransmit the data on the SPI-4 bus in a reprocessed form, and to receive 
data from the host system and retransmit such data on a port. 

[0041] The control bus can be used for programming logic on the port adapter such as 
interface logic, network interfaces and general-purpose registers or other programmable 
elements. The control bus also provides access to control interfaces of devices on the port 
adapter. The power bus is used, in conjimction with software that controls application of 
power, for conducting hot swap operations in which the port adapter is unplugged from or 
plugged into the host system while the host system either is powered and operating or is 
powered down. The port adapter may include a connector with one or more detect pins that 
are shorter than other power bus pins and other signal pins in the connector. In one 
embodiment, the shorter pins are at opposite ends of the connector. The shorter detect pins 
allow the port adapter and host system to anticipate and, in tum, respond to a hot swap 
condition by enabling power to the port adapter only after the port adapter is M\y inserted. 
The shorter pins also enable the host system to determine that all pins are seated correctly. 
Upon removal of a port adapter, the short pins disconnect first and enable the port adapter to 
send signals to the host that are used to disable power to the port adapter. 
[0042] When the port adapter is connected to the host system during a hot swap 
condition, control circuitry starts a controlled power-up sequence. When the port adapter is 

-12- 

50325-0790 (Seq. No. 7683) 



disconnected from the host system during a hot swap condition, the control circuitry starts a 
controlled power-down sequence. 

[0043] A hot swap protocol between the port adapter and the host system discontinues 
data communications on the SPI-4 bus in the port adapter when the port adapter is not at an 
operational power level. When the port adapter is disconnected from the host system, signals 
coming from host circuitry are changed to known safe states to prevent potentially high 
currents from damaging devices during on-line insertion operations. The hot swap protocol 
also prevents corruption of data on the SPI-4 bus and corrupting logic states in the host. 
[0044] A port adapter as disclosed herein can process data that is arriving from interfaces 
up to 10 gigabits per second (Gbps). In other embodiments, improvements to the SPI-4 bus 
architecture that allow faster data rates may be accommodated. 

[0045] FIG. 1 is a block diagram that illustrates an overview of a host system with a port 
adapter. Generally, a host system 100 comprises a central processing unit (CPU) 101 that 
communicates with one or more port adapters 104 A, 104B, 104N.using host interface bus 
102 A, 102B, 102N which are multiplexed through host interface bus hub or multiplexer 109. 
Each host interface bus 102 A, 102B, 102N includes a SPI-4 bus as well as other signals. 
CPU 101 may conununicate with other circuits and devices using one or more other buses 
105, such as an address bus, data bus, etc. For clarity, the depiction of system 100 is greatly 
simplified, and a practical system may include memory devices, I/O devices, route 
processors, network processors, a switch fabric, etc. System 100 maybe implemented as a 
general-purpose packet-switching router or switch. In certain embodiments, system 100 
comprises the Cisco 7300, 7600, 10000, or 12000 series products from Cisco Systems, Inc., 
San Jose, Calif 
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[0046] Host system 100 may have one or more hot-pluggable port adapters 104 A, 104B, 
104N. There may be any number of port adapters in a practical system. A port adapter is also 
referenced herein by the equivalent term "shared port adapter" or SPA, or "versatile port 
adapter" or VP A. Each of the port adapters 104A, 104B, 104N comprises zero or more ports 
106A, 106B, 106N. Each port is communicatively coupled to one of the networks 1 lOA, 
HOB, 1 ION or devices within such networks using any suitable network conununication 
technology, such as Ethemet, Fast Ethemet, Gigabit Ethemet, optical, serial or other 
interfaces. There may be any number of ports on a port adapter in a practical system. Each 
port 106 A, 106B, 106N may be coupled to a different network 1 lOA, 1 lOB, 1 ION. 
[0047] An embodiment with zero ports may comprise a service adapter in which the port 
adapter provides a computational or packet processing service rather than an interface 
function. A port adapter as described herein may have zero ports but may provide, for 
example, an encryption or compression function for the host. Further, in another embodiment 
a combination service adapter and port adapter may be provided that has one or more ports 
and also provides a packet processing service. 

[0048] In the configuration of FIG. 1, each of the port adapters provides a mechanism for 
interfacing its ports to host interface bus 102 A, 102B, 102N, respectively. Each of the port 
adapters 104A, 104B, 104N is hot pluggable, meaning that the port adapters can be removed 
from or installed into host system 100 while the host system is running. As a result, the host 
system 100 can be re-configured with different numbers of ports, or with ports that use 
different network technologies, while retaining the benefits of the SPI-4 bus architecture. 
[0049] FIG. 2 is a block diagram that illustrates an overview of one embodiment of a port 
adapter for the SPI-4 bus. Host system 100, which is omitted in FIG. 2 for clarity, is coupled 
to port adapter 104C through SPI-4 bus 201, control bus 206, extended flow control bus 222, 
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clock bus 224, power control bus 226, and identity bus 228. Port adapter 104C comprises 
SPI-4 termination logic 202, which is communicatively coupled to SPI-4 bus 201 and to a 
network interface 204. In an embodiment in which ports 106A, 106B, 106N are Ethernet 
ports, network interface 204 may be a MAC (media access controller) that is responsible for 
rapidly forming and processing data frames, and can temporarily store data in memory. 
Altematively, network interface 204 may comprise an ATM SAR, etc. Network interface 204 
is communicatively coupled to ports 106 A, 106B, 106N. 

[0050] In the example of FIG. 2, one network interface 204 is shown. In other 
embodiments, a plurality of network interfaces may be provided, and each such network 
interface is coupled to SPI-4 termination logic 202. For example, there may be a different 
network interface 204 for each of the ports 106A, 106B, 106N. 

[0051] In embodiments with zero ports, network interface 204 is omitted and other logic 
for performing packet processing services may be provided in its place. For example, an 
encryption engine or compression engine may occupy the same logical location as network 
interface 204. 

[0052] Control bus 206 is connected to local control logic 208 in port adapter 104C. 
Identity bus 228 is connected to identity element 212, which can be queried by the host 
system 100 to determine the hardware arrangement and logical configuration of the port 
adapter 104C. The port adapter 104C further may include a power control element 214 and 
clock distribution circuit 216 that are respectively coupled to power control bus 226 and 
clock bus 224. 

[0053] Extended flow control bus 222 is coupled to extended flow control logic 220. 
Details of the extended flow control bus are described further in a separate section below. 
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[0054] The port adapter 104C may be implemented as a plurality of integrated circuits 
that are mounted on one or more printed circuit cards that are enclosed in a protective 
housing. In one embodiment, each port adapter is mounted in a slot of a processing circuit 
card in the host system ("host card"). The port adapter housing may have any of several form 
factors, thereby providing a modular arrangement so that multiple different port adapters are 
interchangeable in the same host chassis. In one embodiment, a port adapter housing may 
have a half-height, full-height, double-wide, or high-power form factor based on the number 
and type of ports or interfaces provided in the port adapter, the amount of power dissipated 
by the port adapter, or the area required for the circuitry in the port adapter. 
[0055] Local control logic 208, extended flow control logic 220, clock distribution circuit 
216, power control circuit 214, and identity element 212 are represented in simplified, block 
form for clarity. In particular, connections to each such element are simplified, and each such 
element may have other connections in specific embodiments. Further, port adapter 104C 
may include circuit elements other than the specific elements that are shown in FIG. 2. 
[0056] FIG. 3 is a more detailed block diagram of a port adapter of FIG. 2, according to 
one embodiment. 

[0057] A host connector 302 provides a physical connection to the host system 100 (not 
shown in FIG. 3 for clarity) and carries clock, data, control, and power signals. A 12-volt 
power supply line 226A is coupled to a power conversion block 304 that provides a plurality 
of output power lines 308 at various voltage levels required by other elements of the port 
adapter 104D. In one embodiment, power conversion block provides outputs of 3.3V, 2.5V, 
1.8V, and 1.5 V; other output voltage levels maybe provided in other embodiments. Further, 
power conversion block 304 may provide power sequencing, a power up/down function, 
power monitoring, power margining, etc. 
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[0058] SPI-4 bus 332 is coupled to FPGA 330. The SPI-4 bus 332 typically is an 86-pin 
packet data transfer bus that carries data bus signals, flow control signals, clock signals, etc. 
In certain embodiments, the operating speed of all such signals may be changed for 
compatibility among different hosts and port adapters, as further described herein. A SPA 
bus 206A is coupled from the host system 100 to the FPGA 330. SPA control bus 206A 
enables the host system to control and communicate with internal elements of the port 
adapter 104D. A JTAG bus 307 may carry test signals that are used for testing or PLD 
programming through communication among the host system 100 and programmable 
components; such as an FPGA 330 that implements the functions of SPI-4 termination logic 
202 and local control logic 208 of FIG. 2. Host connector 302 may also carry miscellaneous 
signals for supporting online insertion and removal (OIR) operations, port adapter reset 
functions, etc. 

[G059J FPGA 330 is coupled to fi-amer 204A by a PL3 bus 309, microprocessor bus 3 10, 
and Transport Overhead (TOH) path 312. In one embodiment, framer 204A is the PM5360 
SAJNI Multi-48 SONET/SDH framer, from PMC-Sierra, Inc., Santa Clara, California, which 
provides four (4) ports that are selectable between OC-12/STM-4 and 0C-3/STM-1 
bandwidth. 

[0060] Framer 204A is coupled to one or more small form-factor pluggable (SFP) optics 
modules 314A, 314B, 314C, 314D that provide network ports and interfaces. The FPGA330 
detects insertion or extraction of the modules 314A, 314B, 314C, 314D to or from port 
adapter 104D. 

[0061] Generally, FPGA 330 functions to decode and interface signals of the SPA bus 
206A to signals from microprocessor bus 310. The FPGA 330 also provides control and 
status inforaiation relating to SFPs 314A, 314B, 314C, 314D. Further, the FPGA 330 
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provides bridging, queuing, and scheduling for communications among the PL3 bus 309 and 
SPI-4 bus 332, including management of ingress and egress FIFO queues, and the FPGA 330 
may be involved in online insertion and removal and power control functions. The FPGA 
330 is also configured for inserting and extracting SONET overhead information fi-om 
packets that are communicated to or fi*om the framer 204A. In one embodiment, FPGA 330 
is implemented using the Xilinx 2V1500 and the SPI4, PL3, and HDLC IP cores. 
[0062] Port adapter 104D also may include various other functional elements including 
clock generation/recovery module 216A, identity electrically erasable programmable read- 
only memory ("ID EEPROM") 21 2A, voltage supervisor 228 A, temperature sensors 320, and 
voltage margining unit 322. The clock generation/recovery module 216A receives a 77.76 
MHz SONET reference clock 224B from the host through host connector 302, provides a 
recovered 19.44 MHz clock 224A to the host via host connector 302, provides a 77.76 MHz 
reference clock 224C to the firamer 204A, and receives a recovered 77.76 MHz clock signal 
224D firom the fi-amer The use of a recovered clock enables the clock generation/recovery 
module 216A to derive a system clock from any attached SONET port. The clock 
generation/recovery module 21 6A also generates a 100 MHz clock for operating FPGA 330. 
[0063] The temperature sensors 320 may have a programmable temperature range for 
detecting and signaling over-temperature problems. 

[0064] Optionally, an extended flow control bus coupled from host system 100 to the 
port adapter 104D functions to provide back pressure for port adapters with very high counts 
of physical or virtual ports. For example, in ATM, numerous virtual circuits may be present 
on one physical link. Thus, the extended flow control bus may be used for highly channelized 
port adapters, ATM port adapters, etc. 
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[0065] A detailed specification for an embodiment of a port adapter is provided in the 
Appendix to this document, the entire contents of which are hereby incorporated by reference 
as if fully set forth herein. 

2.2 ADAPTATION TO DIFFERENT SPI-4 OPERATING SPEED RATES 
[0066] The host system 100 can query the ID EEPROM 212A using an identity bus 228A 
to determine the configuration of the port adapter 104D and to perform power control 
functions. In one embodiment, bus 228A conforms to the I2C signal format. The ID 
EEPROM 212A is an example of an identification repository . Based on the identifying 
information, software executed by the host system determines values specifying an operating 
frequency of the port adapter, and the format of data that is sent across the SPI-4 bus. For 
example, the host system software may include a lookup table that maps bus speed values, 
data packet formats, etc., to various port adapter identifiers. In an alternate embodiment, the 
identification repository stores bus speed values, data packet formats, and other configuration 
parameters in association with one or more port adapter identifiers for that port adapter or 
several different port adapters. 

[0067] The information in the identification repository enables the host system to adapt 
its operational behavior to particular characteristics of the port adapter or its ports. For 
example, the standard operating fi-equency of the SPI-4 bus is 350 MHz ("full rate SPI-4"). 
However, not all port adapters require this frequency. For example, a port adapter that 
supports an aggregate data communication bandwidth of greater than 2.4 Gbps on its 
interfaces may require a full rate SPI-4 bus, but other port adapters that support only 
aggregate data conmiunication bandwidth of less than or equal to 2.4 Gbps may operate 
adequately using SPI-4 bus signaling at less than 350 MHz. 
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[0068] Therefore, in one embodiment, the SPI-4 bus of port adapter 104D may be 
configured to operate at a quarter-rate speed of 87.5 MHz. In other embodiments, the SPI-4 
bus of port adapter i04D may be configured to operate at any other speed, e.g., 700 MHz 
providing double-rate speed, etc. The identification repository of a port adapter contains a 
port adapter type identifier. Based on the port adapter type identifier, software executed by 
the host system can detemiine whether the port adapter supports a full rate SPI-4 bus speed, 
quarter rate, or both, or some other speed. Generally, in one embodiment, 

L A port adapter 104D that supports a total bandwidth less than or equal to 2.4 

Gbps on its interfaces must support quarter rate on its SPI-4 bus, and may also 

optionally support fiill rate; 

2. A port adapter that supports a total bandwidth of greater than 2.4 Gbps on its 
interfaces must support full rate on its SPI-4 bus, and may optionally also 
support quarter rate. 

Host systems should conform to similar rules to ensure bandwidth compatibility across the 
SPI-4 connection to a port adapter. Thus, 

3. A host system that supports a bandwidth of less than or equal to 2.4 Gbps in 
any of shared port adapter slot must support quarter rate on the SPI-4 bus for 
that slot, and may also optionally support full rate; 

4. A host system that supports bandwidth of greater than 2.4 Gbps in any slot 
must support full rate on the SPI-4 bus for that slot, and may optionally also 
support quarter rate. 

[0069] Using this arrangement, the host system may query the identification repository 
and adapt its operational behavior based on information in the identification repository. 
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[0070] FIG. 4A is a flow diagram of a process of adapting the operational behavior of the 
SPI-4 bus of a host system based on a capability of a port adapter. In block 402, a query is 
issued to an identification repository in a port adapter. For example, with reference to FIG. 3, 
host system 100 may issue signals on bus 228 A to read the contents of ID EEPROM 212A. 
Block 402 may be performed before the host system provides power to a port adapter such as 
port adapter 104D. An identification repository such as ID EEPROM 212A may receive 
power from a separate power pin in the connector 302, which enables host system 100 to 
read port configuration information from the port adapter even when the port adapter is 
powered off 

[0071] In block 404, a response is received from the port adapter that includes a unique 
identifier for the port adapter. For example, reading ID EEPROM 212A results in port 
adapter 104D providing its unique identifier value. In block 405, the host determines one or 
more SPI-4 bus operating rates that are supported by the port adapter. For example, the host 
uses a stored lookup table to associate the received unique identifier value with one or more 
operating frequency values for the port adapter. Additionally, the host system may determine 
whether the port adapter can be supported by the host system, and the format of data that is 
sent by the port adapter on the SPI-4 bus. Alternatively, such values and configuration 
parameters are provided from the identification repository of the port adapter. 
[0072] In block 406, the host system determines whether it is compatible with one of the 
supported rates that the host determined based on the identifier received from the 
identification repository of the port adapter. Block 406 may involve applying rules 1-4 as 
denoted above to determine whether a port adapter and host are compatible. For example, if 
the port adapter supports only quarter rate SPI-4, and the host requires full rate, then the host 
is not compatible with the port adapter. If the host is not compatible, then in block 407, the 
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host does not power-up the port adapter, which cannot be used by the host system. The rules 
1-4 above may be implemented in software executed by the host system. 
[0073] Optionally, the process involves powering-on the port adapter only when one or 
more factors are satisfied. For example, block 406 can involve evaluating factors such as: 
whether the host system has software support for a packet format required by the port 
adapter; whether the host system has software support for the port adapter; whether the port 
adapter dissipates less than a maximum amount of power dissipation allowed by the host 
system; whether the host system can match a bandwidth required by the port adapter; 
whether a license authorization requirement associated with the port adapter allows the port 
adapter to run on the host system; etc. 

[0074] If the host is compatible with the port adapter, then in block 408 the host changes 
the operating rate of its own SPI-4 bus to the fastest compatible supported rate. For example, 
if the port adapter identification repository indicates that the port adapter supports both 
quarter rate and fiiU rate SPI-4, then the host changes its SPI-4 operating rate to fiill rate. In 
an altemative embodiment, the host changes the operating rate of its own SPI-4 bus to any 
one of the compatible supported rates. 

[0075] In block 410, the host powers-up the port adapter by sending appropriate control 
signals; in the example of FIG. 2, such signals may be sent on the power control bus 226. In 
block 412, the host sets the port adapter to the same compatible rate that the host is using. For 
the example of FIG. 3, host system 100 sends control signals on SPA bus 206 A to instruct 
the port adapter to use a particular rate. The rate that is set may be the fastest compatible rate, 
or any selected compatible rate. 

[0076] Optionally, in other embodiments, the order of performing steps 408, 410, 412 
may be changed, and the order of performing such steps is not critical. 
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[0077] Thus, using the approach of FIG. 4 A, a host can query a port adapter for 
information about operational characteristics of the port adapter, and based on the received 
information, the host system determines whether it is compatible, whether to power-on the 
port adapter, and what operating rate to use. 

[0078] Further, based on the received information, the host system may determine an 
operating frequency at which to run the SPI-4 bus. For example, full rate SPI-4 may be used, 
quarter rate may be used, etc. 

[0079] The host system may also determine a particular format for data conununication 
on the SPI-4 bus, as described further in section 2.4 below, for example. 

2.3 EXTENDED FLOW CONTROL BUS 
[0080] A conventional SPI-4 bus addresses a maximum of 256 channels, and provides 
support for FIFO queue status indications for 256 chaimels in a normal addressing mode. 
However, port adapters that have a large number of channels ("highly channelized" or ATM 
SPAs, for example) may need 1,000 or more channels. Therefore, it is desirable to have a 
port adapter flow control bus that can support more than 256 channels per port adapter. 
[0081] Accordingly, an extended flow control bus and associated method is provided to 
extend a port adapter to enable use with more than 256 channels. In this arrangement, a port 
adapter requiring less than or equal to 256 channels may use a conventional SPI-4 control 
bus for flow control, and optionally may use an extended flow control bus as defined herein. 
If a port adapter uses the extended flow control bus as defined herein, the port adapter also 
still uses the conventional SPI-4 flow control bus for gross (rather than subchannel or virtual 
channel) flow control of traffic aggregates such as port adapter-level or physical port-level 
flow control. 
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[0082] In one embodiment, Extended Flow Control Bus 222 carries a time domain 
multiplexed (TDM) calendar frame sync signal, a flow control clock signal, a status signal, 
and a parity signal. The flow control clock signal provides a source clock that is used by the 
host to clock in the data value on the status signal, and is sourced by the sender of flow 
control data, which is normally the port adapter. An example clock frequency is 50 MHz, but 
any other suitable clock frequency may be used. 

[0083] In one embodiment, the status signal is a one-bit signal, but other forms of status 
signaling may be used. The status signal provides an indication whether channel FIFO status 
is above or below a threshold value, corresponding to the channel programmed for the TDM 
timeslot. The parity value provides even or odd parity, in various embodiments, across the 
status signal and frame sync signal for a particular clock cycle. Use of a separate parity signal 
allows flexibility in changing the frame size to any length, in various embodiments. 
Optionally, a port adapter may not support the extended flow control bus, in which case the 
foregoing signals are not connected. 

[0084] Thus, in an embodiment, the extended flow control bus uses a TDM calendar- 
based mechanism that carries per-channel FIFO status information over a single data bit. The 
calendar is programmed by the host system 100 when channels are configured and set up at 
the port adapter and host. In one embodiment, time slots are allocated in proportion to the 
bandwidth of the channel. Embodiments may approximate channel bandwidths to the closest 
power of 2 and may allocate time slots in a way that reduces the total number of flow control 
time slots. In one embodiment, the calendar comprises a table in which rows correspond to 
timeslots and columns carry channel numbers and FIFO status information. In one particular 
embodiment, there are 16584 rows each comprising a channel number in 12 bits and one 
status bit. 
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[0085] The port adapter uses the calendar to determine which channel is polled for FIFO 

status and which channel is sent in a particular timeslot or clock period. The host uses a 
similarly configured calendar to determine which channel's FIFO status flow control 
information is carried in a particular timeslot. 

[0086] In one embodiment, the number of supported channels is configurable so that it 
can adjust to the capabilities of a particular host. For example, a host card may support only 
IK flow-controllable entities, and therefore certain port adapters may need to support fewer 
than the maximum number of channels. 

[0087] A detailed description of the extended flow control bus is provided in section 2.3 
of the Appendix. 

2.4 PRE-PROCESSING PACKETS WITH PORT ADAPTER 
[0088] In one embodiment, each port adapter 104 A, 104B, 104N may communicate with 
external networks or devices using any of a large number of network technologies. As a 
result, data packets that are received at the port adapters may have any of a large number of 
different formats. In an embodiment, each port adapter provides data to the host in one of a 
small number of basic packet formats, all of which are understood by the host. For example, 
in one specific embodiment, four (4) packet formats are used, and a port adapter supports one 
or more of the four formats to communicate with a host. In this approach, since port adapters 
are targeted at many different host systems, the formats hide the detail and processing burden 
associated with a specific media type as much as possible within the port adapter to assist the 
host to operate at high speed or with less complex packet processing. In addition, the packet 
formats provide header fields that are as small as possible, to reduce the bandwidth utilized 
on the SPI-4 bus. 
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[0089] FIG. 4B is a flow diagram that illustrates an overview of a process of 
transforming received data packets. 

[0090] In block 422, a packet is received on an ingress interface of a port adapter. In one 
embodiment, the process of FIG. 4B is performed by a port adapter as shown in FIG. 2. Thus, 
the steps of FIG. 4B may be performed by the SPI-4 termination logic 202, for example. The 
packet received at block 422 is formatted according to a native packet format of a particular 
networking technology that is supported by the port adapter. Example technologies include 
Ethemet, ATM, Frame Relay, etc. 

[0091 J In block 424, one of a pluraUty of different packet formats is selected. Block 424 
typically involves selecting one of several packet formats, e.g., a format other than the native 
format in which the packet was received. In one specific embodiment described further 
below, a packet format is selected from among Ethemet SPA 8-byte shim format, ATM SPA 
4-byte shim format. Highly Channelized SPA 4-byte shim format, and a no shim format. The 
selected format may include more or fewer data fields than the fields that are in the received 
packet. Performing block 424 may comprise simply selecting one specified packet format 
associated with the then-current port adapter. Further, in the case of a port adapter that 
supports Ethemet packets, a particular packet format may be selected based on a VLAN 
identifier carried in a packet. 

[0092] In block 426, a new packet conforming to the selected format is created. 
[0093] In block 428, data from the fields of the received packet is transformed into one or 
more corresponding fields of the new packet. The data transformation may be performed 
according to a data-driven mapping or programmatic rules that specify which fields of a 
particular ingress packet format are transformed to which other fields of the target packet 
format. Further, the mapping or rules may specify transformations of data or values obtained 
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from sources other than the packet, such as interface identifier, packet length, congestion 

status, packet validity checks, etc. 

[0094] In block 430, the new packet is provided to the host system. For example, in FIG. 
2, the new packet is communicated from SPI-4 termination logic 202 on the host system bus 
201 to the host system 100. 

[0095] The four packet formats used in an embodiment may be designated, for example, 
as: 

Format A: Ethemet SPA 8 byte shim format. 
Format B: ATM SPA 4 byte shim format. 
Format C: Highly Channelized SPA 4 byte shim format. 
Format D: No shim format 
Each such format is described in detail in the Appendix. 

[0096] FIG. 5 is a block diagram of a generalized transformed packet format. In the 
embodiment of FIG. 5, a packet 500 comprises classification bits 502, length indicator 504, 
source channel label 506, and header fields 508. The classification bits 502 carry information 
conveying a class value associated with the received packet. The classification information 
may originally derive from any of several different packet fields, including but not limited to 
the Type of Service (ToS) field of an IP packet, 802. Iq priority information, MAC address 
filtering information, etc. Length indicator 504 may specify a length adjustment that has been 
made to the original packet, or may specify an absolute length of the transformed packet. 
Source channel label 506 specifies a logical or physical channel on which the original packet 
arrived. Header fields 508 carry information derived from header fields of the original 
packet. 
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[0097] The generalized format of FIG. 5 may be adapted in various ways to different 
formats of inbound packets. For example, in an Ethemet SPA 8 byte Shim Format, the port 
adapter strips Layer 2 encapsulation from a packet entirely and replaces it with an 8-byte 
shim header that includes all relevant information from the original packet for a forwarding 
engine of the host to make an efficient forwarding decision. The lower 4 bytes are 
approximately formatted in the same way as a Frame Relay header, allowing possible 
simplification of design of the host's forwarding engine. 

[0098] Stripping the Layer 2 header is optional on a per-packet basis, allowing support 
for Layer 2 tunnels such as Ethemet over MPLS. If the Layer 2 header is left on the packet, 
then it can also optionally be padded with two or three bytes to bring the Layer 3 header to 4- 
byte alignment, as an optional optimization for some hosts. The first byte of the padding 
indicates the number of padding bytes present, for example. 

[0099] In this case, because the format of the packet leaving the port adapter can include 
optional stripping of the variable-length Layer 2 encapsulation, and the addition of a shim 
header, the Length Indicator value 504 indicates the number of bytes by which the packet is 
shorter as compared to when the packet was first received. The Layer 3 engine of host system 
100 can determine the original Layer 2 length by adding the value of Length Indicator value 
504 to the total number of bytes received from the port adapter. 

[0100] Header fields 508 may include the Protocol ID ("PID") of the Layer 2 header of 
the packet, and the port adapter may have translated the value. Certain special values of the 
PID field indicate that the host must apply special treatment to the particular packet; the 
special values are sofl:ware configurable. For example, special PIDs may be used to indicate a 
tunneled packet, exception packet, or other special characteristics. For a tunneled packet, 
when the VLANED and port number of the arriving packet are configured to enter an Layer 2 
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tunnel, then the entire packet with its original Layer 2 encapsulation is brought into the host 
system. An exception packet indicates that the port adapter has detected something about the 
packet that requires the host to perform special treatment on the packet. More than one 
exception packet special PID may be defined. This may allow classification of the packets 
into different priority CPU queues, for example. 

[0101] Optionally, as part of transforming a first packet into a particular selected packet 
format, the header 5 1 0 and/or body 5 12 of the original packet may be placed in the 
transformed packet 500. Thus, the packet format used within the host and port adapter may 
include the original packet header 510 and/or original packet body 512. The original header 
and body may be omitted depending on the nature of the traffic that is processed or the 
context in which it is processed. . . 

[0102] Similar transformation techniques may be applied to other different packet 
formats of inboxmd packets. 

[0103] Additionally or alternatively, rather than transforming packets, packets may be 
dropped. For exaniple, if a port adapter receives a packet from a Layer 2 address or VLAN 
that is of no interest to the port adapter or host, then that packet may be dropped. 
3.0 EXTENSIONS AND ALTERNATIVES 

[0104] In the foregoing specification, the invention has been described with reference to 
specific embodiments thereof It will, however, be evident that various modifications and 
changes may be made thereto without departing fi-om the broader spirit and scope of the 
invention. The specification and drawings are, accordingly, to be regarded in an illustrative 
rather than a restrictive sense. 

[0105] For example, a port adapter may classify packets as high priority or low priority 
and provide priority information in the transformed packet format to enable the host to 
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determine which packets to process first. As an alternative to carrying packet priority 
information in the transformed packet format, two or more logical SPI-4 chaimels may be 
associated with one physical port, in which a first logical channel carries port traffic 
associated with a first priority level and the second or additional channel(s) carry traffic 
associated with a second or other priority level. In this approach, the host adapter may be 
configured through software to process all packets arriving on the higher priority chaimel 
first without having to consult priority information within the packets. 
[0106] In another variation of the architecture described above, one of the logical SPI-4 
channels may be used as a control path as an alternative to providing some or all control 
signals on control bus 206 (FIG. 2) or SPA bus 206A (FIG. 3). In this alternative, a specified 
logical charmel carries control packets separate fi-om an associated logical chaimel that 
carries port data traffic. The control chaimel may be considered as having the highest priority 
for the host. Further, an advantage of this approach is that the control packets may be 
synchronized or aligned in time with the transmission of associated data packets. Further, the 
bandwidth of the SPI-4 bus used for the logical channel used for control in this approach is 
typically higher than the bandwidth of the SPA bus 206A or control bus 206, which may be 
usefiil for sending large volumes of control data, statistics, etc. 

[0107] In still another variation, a specified SPI-4 logical channel can be used to carry 
flow control information, e.g., in the form of events. 
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